So-called MR (Mixed Reality) techniques are known recently as techniques of seamlessly merging a real world and a virtual world in real time. In a known MR technique, a video see-through type HMD (Head-Mounted Display) is used. An object which is almost coincident with an object observed from the pupil position of an HMD wearer is taken by, e.g., a video camera. The HMD wearer can observe an image obtained by superimposing CG (Computer Graphics) on the taken image.
FIG. 3 is a view showing the schematic arrangement of a general video see-through type HMD. The video see-through type HMD includes a pair of structures for right and left eyes. FIG. 3 illustrates only one structure. The HMD includes a display device 201 such as a right or left small liquid crystal display, a display optical system 202 such as a free-form surface prism to zoom and display a right or left image displayed on the display device, an image pickup device 301 to take an object which is almost coincident with an object observed from the position of a pupil 401 of an HMD wearer, and an image taking optical system 302 to make the position of the pupil 401 of the HMD wearer almost coincident with the position of the image pickup device 301. FIG. 4 shows the outer appearance of a video see-through type HMD 501 worn by a user.
FIG. 5 is a block diagram showing the main hardware configuration of a video see-through type HMD 500. The video see-through type HMD 500 includes image taking units 110R and 110L which take the external world, display units 120R and 120L which display a composited image of CG generated by an external image processing unit 140 and taken images obtained from the image taking units 110R and 110L, and an MPU 130 which controls the image taking units 110R and 110L and display units 120R and 120L.
The image taking units 110R and 110L include image pickup devices 111R and 111L such as CCD image sensors, TGs (Timing Generators) 116R and 116L which generate signals to drive the image pickup devices 1l1R and 111L, V-Drs (V Drivers) 117R and 117L which receive the signals from the TGs 116R and 116L and generate vertical signals, CDS/AGC units 112R and 112L which execute processing such as CDS (Correlated Double Sampling) and gain adjustment for analog signals output from the image pickup devices 111R and 111L, A/D conversion units 113R and 113L which convert the analog signals into digital signals, digital signal processing units 114R and 114L which execute gain control, tint control, luminance control, and gamma correction for the digital signals, and taken image output units 115R and 115L which convert the taken image signals output from the digital signal processing units 114R and 114L into an image format for, e.g., USB or IEEE 1394 and output the image signals to the image processing unit 140.
The display units 120R and 120L include display devices 121R and 121L such p-SiTFT or LCOS, display driving units 122R and 122L to control and drive the display devices 121R and 121L, and display image input units 123R and 123L which receive display image signals output from the image processing unit 140.
The image processing unit 140 is composed of a computer such as a PC (Personal Computer) or WS (WorkStation). The image processing unit 140 generates CG, composites it on a taken image received from the video see-through type HMD 500, and returns the image to the video see-through type HMD 500 again.
With the above-described arrangement, the HMD wearer can experience an MR world, in which the real world and virtual world are seamlessly merged in real time, by wearing the video see-through type HMD on his/her head.
The left and right image sensing units incorporated in the general video see-through type HMD have independent structures. For this reason, images taken by the left and right image taking units are not completely identical, and the left and right image taking conditions do not always match. For example, when AE (Auto Exposure processing) and AWB (Auto White Balance processing) are executed for each of the left and right image taking units, image taking conditions such as the analog gain, shutter speed, and RGB gain may not match between the left and right units, and the left and right taken images may have different brightness and tint levels.
As another problem, even when a locally bright image is taken on only one image taking screen, or only one image taking screen is covered with, e.g., a hand, the left and right images may have different brightness and tint levels.